Skip to content

feat(scout): give the on-host discovery agent a /metrics endpoint#3539

Open
chet wants to merge 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3173-scout
Open

feat(scout): give the on-host discovery agent a /metrics endpoint#3539
chet wants to merge 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3173-scout

Conversation

@chet

@chet chet commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

scout -- the on-host control plane -- was invisible to Prometheus: no metrics and no /metrics endpoint, only logs. This stands one up through the shared metrics-endpoint crate and counts the control loop's work.

  • With --metrics-listen-addr set, scout installs the framework meter, registers carbide_log_events_total (so the existing error/warn sites gain an error-rate signal for free), and serves /metrics. Without the flag scout stays a pure client as before -- the endpoint is opt-in, and the counters record nothing until it is configured.
  • carbide_scout_actions_total{action, outcome} counts each control-loop action by kind and result; carbide_scout_stream_connections_total{outcome} and carbide_scout_stream_reconnects_total track the ScoutStream lifecycle.
  • A failed log-error action now propagates instead of being swallowed, so its outcome records as an error rather than a silent success.
  • The existing error!/warn! lines stay as logs -- their rate rides on carbide_log_events_total, so there are no per-site metric siblings.

scout's /metrics is binary-local (not scraped by test_integration), so there is no catalogue change.

This supports #3173

@chet

chet commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@chet chet requested a review from a team as a code owner July 15, 2026 06:33
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@chet, sounds good — I'll redo the full review of the changes now.

ᕮ(•ᴥ•)ᕭ

✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Summary by CodeRabbit

  • New Features
    • Added an optional command-line setting to enable an HTTP metrics/health endpoint (and start collecting metrics).
    • Added new Prometheus-style metrics for scout action handling, stream connection outcomes, and reconnect attempts.
  • Bug Fixes
    • Log-error handling failures are now surfaced rather than being suppressed.
  • Tests
    • Added unit tests to ensure action-to-metric label mapping covers all variants and emitted counters change as expected.

Walkthrough

Scout adds an optional metrics/health endpoint, defines Prometheus-style action and stream metrics, and emits lifecycle and action outcomes. Log-error conversion failures are now propagated instead of only being logged.

Changes

Scout metrics instrumentation

Layer / File(s) Summary
Metric contracts and configuration
crates/scout/Cargo.toml, crates/scout/src/cfg/command_line.rs, crates/scout/src/metrics.rs
Adds metrics dependencies, the optional metrics_listen_addr CLI option, bounded action labels, stream counters, and tests for mappings and counter increments.
Service metrics and action instrumentation
crates/scout/src/main.rs
Conditionally starts the metrics endpoint, emits action outcomes, and propagates LogError conversion failures.
Stream lifecycle instrumentation
crates/scout/src/stream.rs
Emits connection success/error counters and reconnect-cycle counters during stream processing.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Options
  participant ScoutService
  participant MetricsEndpoint
  participant ActionLoop
  participant carbide_instrument
  Options->>ScoutService: metrics_listen_addr
  ScoutService->>MetricsEndpoint: start metrics endpoint
  ActionLoop->>ActionLoop: handle_action
  ActionLoop->>carbide_instrument: emit ScoutActionHandled with action and outcome
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding an opt-in /metrics endpoint for Scout.
Description check ✅ Passed The description accurately describes the Scout metrics endpoint, new counters, and related error-handling changes.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
crates/scout/src/main.rs (1)

194-198: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use structured fields for endpoint logs.

Line 195 and Line 197 interpolate queryable values into messages. Emit address and error as tracing fields instead.

Proposed fix
- tracing::info!("Spawning metrics endpoint on {}", metrics_config.address);
+ tracing::info!(address = %metrics_config.address, "spawning metrics endpoint");
  if let Err(e) = metrics_endpoint::run_metrics_endpoint(&metrics_config).await {
-     tracing::error!("Metrics endpoint error: {e}");
+     tracing::error!(error = %e, "metrics endpoint error");
  }

As per coding guidelines, “Use plain tracing:: macros with structured fields for ordinary logging.” As per path instructions, use “structured tracing fields.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/scout/src/main.rs` around lines 194 - 198, Update the metrics endpoint
logs inside the tokio::spawn closure to use structured tracing fields: emit
metrics_config.address as an address field in the info event and the endpoint
failure as an error field in the error event, while keeping the existing log
messages and behavior.

Sources: Coding guidelines, Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/scout/src/stream.rs`:
- Around line 92-97: Update the stream-establishment error path around
client.scout_stream(request_stream).await so it emits ScoutStreamConnection with
Outcome::Error before propagating the failure. Preserve the existing
reconnect-loop behavior and the create_forge_client metric handling.

---

Nitpick comments:
In `@crates/scout/src/main.rs`:
- Around line 194-198: Update the metrics endpoint logs inside the tokio::spawn
closure to use structured tracing fields: emit metrics_config.address as an
address field in the info event and the endpoint failure as an error field in
the error event, while keeping the existing log messages and behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bbf53f66-4e05-4c5e-b39c-1617db7bf117

📥 Commits

Reviewing files that changed from the base of the PR and between eb4ea22 and 68f3326.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (5)
  • crates/scout/Cargo.toml
  • crates/scout/src/cfg/command_line.rs
  • crates/scout/src/main.rs
  • crates/scout/src/metrics.rs
  • crates/scout/src/stream.rs

Comment thread crates/scout/src/stream.rs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
crates/scout/src/cfg/command_line.rs (1)

111-117: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Type the metrics listen address as SocketAddr at the CLI boundary. Both sites treat the metrics address as a bare string, pushing validation from CLI parsing into runtime service startup instead of letting clap/the compiler catch a malformed address immediately.

  • crates/scout/src/cfg/command_line.rs#L111-L117: change pub metrics_listen_addr: Option<String> to pub metrics_listen_addr: Option<SocketAddr>; clap derives parsing from SocketAddr's FromStr automatically, giving a clear CLI-level error on malformed input.
  • crates/scout/src/main.rs#L179-L182: drop the manual addr_str.parse() / eyre::eyre!("Invalid --metrics-listen-addr...") block and use config.metrics_listen_addr directly as the already-typed SocketAddr.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/scout/src/cfg/command_line.rs` around lines 111 - 117, The metrics
listen address is validated too late as a string. In
crates/scout/src/cfg/command_line.rs:111-117, change CommandLine’s
metrics_listen_addr field to Option<SocketAddr> and ensure the SocketAddr type
is imported; in crates/scout/src/main.rs:179-182, remove the manual
addr_str.parse() and custom invalid-address error handling, passing
config.metrics_listen_addr directly to the metrics/health endpoint startup.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/scout/src/stream.rs`:
- Around line 118-120: Update the client.scout_stream call in the
response-stream setup to record ScoutStreamConnection { outcome: Outcome::Error
} before propagating any handshake failure. Preserve successful stream
initialization and the existing error propagation behavior, while ensuring both
create_forge_client and scout_stream failures increment the connection error
metric.

---

Nitpick comments:
In `@crates/scout/src/cfg/command_line.rs`:
- Around line 111-117: The metrics listen address is validated too late as a
string. In crates/scout/src/cfg/command_line.rs:111-117, change CommandLine’s
metrics_listen_addr field to Option<SocketAddr> and ensure the SocketAddr type
is imported; in crates/scout/src/main.rs:179-182, remove the manual
addr_str.parse() and custom invalid-address error handling, passing
config.metrics_listen_addr directly to the metrics/health endpoint startup.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a92e7ba3-9717-421a-8964-79859ad1ab41

📥 Commits

Reviewing files that changed from the base of the PR and between eb4ea22 and 68f3326.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (5)
  • crates/scout/Cargo.toml
  • crates/scout/src/cfg/command_line.rs
  • crates/scout/src/main.rs
  • crates/scout/src/metrics.rs
  • crates/scout/src/stream.rs

Comment thread crates/scout/src/stream.rs
scout -- the on-host control plane -- was invisible to Prometheus: no metrics and no /metrics endpoint, only logs. This stands one up through the shared metrics-endpoint crate and counts the control loop's work.

- With `--metrics-listen-addr` set, scout installs the framework meter, registers `carbide_log_events_total` (so the existing error/warn sites gain an error-rate signal for free), and serves `/metrics`. Without the flag scout stays a pure client as before -- the endpoint is opt-in, and the counters record nothing until it is configured.
- `carbide_scout_actions_total{action, outcome}` counts each control-loop action by kind and result; `carbide_scout_stream_connections_total{outcome}` and `carbide_scout_stream_reconnects_total` track the ScoutStream lifecycle.
- A failed log-error action now propagates instead of being swallowed, so its outcome records as an error rather than a silent success.
- The existing `error!`/`warn!` lines stay as logs -- their rate rides on `carbide_log_events_total`, so there are no per-site metric siblings.

scout's /metrics is binary-local (not scraped by `test_integration`), so there is no catalogue change.

This supports NVIDIA#3173

Signed-off-by: Chet Nichols III <chetn@nvidia.com>
@chet chet force-pushed the gh-issue-3173-scout branch from 68f3326 to e2ee040 Compare July 15, 2026 07:44
@chet

chet commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @coderabbitai -- both addressed:

  • A scout_stream handshake failure now also emits carbide_scout_stream_connections_total{outcome="error"}, so both the client-create and the stream-handshake failure paths count (successful init still records ok).
  • --metrics-listen-addr is now Option<SocketAddr>, parsed and validated by clap at the arg layer; the late manual parse in main.rs is gone.

@chet

chet commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@chet, sounds good — I'll redo the full review of the changes now.

✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
gh pr view 3539 --repo NVIDIA/infra-controller --json title,body,commits,files -q '.files[].path'
echo "---diff stream.rs---"
gh pr diff 3539 --repo NVIDIA/infra-controller -- crates/scout/src/stream.rs
echo "---diff command_line.rs---"
gh pr diff 3539 --repo NVIDIA/infra-controller -- crates/scout/src/cfg/command_line.rs
echo "---diff main.rs---"
gh pr diff 3539 --repo NVIDIA/infra-controller -- crates/scout/src/main.rs

Length of output: 1074


🏁 Script executed:

#!/bin/bash
gh pr diff 3539 --repo NVIDIA/infra-controller > /tmp/pr3539.diff
echo "---stream.rs---"
awk '/^diff --git a\/crates\/scout\/src\/stream.rs/,/^diff --git a\/crates\/scout\/src\/[^s]/' /tmp/pr3539.diff | sed '$d'
echo "---command_line.rs---"
awk '/^diff --git a\/crates\/scout\/src\/cfg\/command_line.rs/,0' /tmp/pr3539.diff | sed -n '1,80p'
echo "---main.rs relevant---"
grep -n "metrics_listen_addr\|SocketAddr" /tmp/pr3539.diff

Length of output: 6310


@chet, confirmed both changes look correct:

  • In crates/scout/src/stream.rs, the client.scout_stream(...) call now maps its error through emit(ScoutStreamConnection { outcome: Outcome::Error }) before converting via ScoutStreamError::from(e), alongside the existing create_forge_client failure path — so carbide_scout_stream_connections_total{outcome="error"} is incremented on either failure mode, and outcome="ok" is still recorded only after into_inner() succeeds.
  • In crates/scout/src/cfg/command_line.rs, metrics_listen_addr is now Option<std::net::SocketAddr>, so clap validates the address at parse time; main.rs consumes config.metrics_listen_addr directly without any late manual parsing.

Both look solid. I'll kick off the full review now.

ᕮ(•ᴥ•)ᕭ ~ happy to see this landed cleanly

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/scout/src/stream.rs (1)

92-97: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Minor duplication in the emit-then-map-error pattern.

The emit(ScoutStreamConnection{outcome: Outcome::Error}); ScoutStreamError::X(...) sequence is repeated verbatim across both error paths. A tiny helper (e.g., a closure factory or a record_connection_error fn) could collapse this, though with only two call sites the current inline form is still perfectly readable.

♻️ Optional consolidation
+fn record_connection_error<E: Into<ScoutStreamError>>(e: E) -> ScoutStreamError {
+    emit(ScoutStreamConnection {
+        outcome: Outcome::Error,
+    });
+    e.into()
+}
+
 async fn run_scout_stream_loop(
     machine_id: MachineId,
     options: &Options,
 ) -> Result<(), ScoutStreamError> {
-    let mut client = client::create_forge_client(options).await.map_err(|e| {
-        emit(ScoutStreamConnection {
-            outcome: Outcome::Error,
-        });
-        ScoutStreamError::ClientError(e.to_string())
-    })?;
+    let mut client = client::create_forge_client(options)
+        .await
+        .map_err(|e| record_connection_error(ScoutStreamError::ClientError(e.to_string())))?;

Also applies to: 119-128

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/scout/src/stream.rs` around lines 92 - 97, Optionally consolidate the
repeated connection-error handling in the client creation and subsequent error
paths around create_forge_client and the second map_err block by introducing a
small shared helper or closure that emits ScoutStreamConnection with
Outcome::Error and constructs the corresponding ScoutStreamError. Replace both
inline emit-then-map-error sequences while preserving each path’s existing error
variant and message.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/scout/src/stream.rs`:
- Around line 92-97: Optionally consolidate the repeated connection-error
handling in the client creation and subsequent error paths around
create_forge_client and the second map_err block by introducing a small shared
helper or closure that emits ScoutStreamConnection with Outcome::Error and
constructs the corresponding ScoutStreamError. Replace both inline
emit-then-map-error sequences while preserving each path’s existing error
variant and message.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 698b164e-26db-4189-84f7-334ae9237572

📥 Commits

Reviewing files that changed from the base of the PR and between eb4ea22 and e2ee040.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (5)
  • crates/scout/Cargo.toml
  • crates/scout/src/cfg/command_line.rs
  • crates/scout/src/main.rs
  • crates/scout/src/metrics.rs
  • crates/scout/src/stream.rs

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

@chet, thanks for confirming — both changes are already verified in my earlier pass on this thread:

  • crates/scout/src/stream.rs: the scout_stream handshake failure now emits Outcome::Error via emit(ScoutStreamConnection { outcome: Outcome::Error }) before converting to ScoutStreamError::from(e), matching the create_forge_client failure path. outcome="ok" still only fires after into_inner() succeeds.
  • crates/scout/src/cfg/command_line.rs: metrics_listen_addr is Option<std::net::SocketAddr>, so clap owns parsing/validation, and main.rs no longer does any manual late parsing.

Nothing further needed from my side here.

ᕮ(•ᴥ•)ᕭ

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
crates/scout/src/metrics.rs (1)

77-90: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Doc comment undersells the error outcome's coverage.

The comment says error fires only "when the client could not be built," but stream.rs now also emits Outcome::Error on client.scout_stream(...) handshake failures (a separate, later failure point). Worth updating the doc so operators reading this metric's semantics aren't misled about which failures are counted.

✏️ Proposed doc fix
-/// A scout stream connection attempt resolved -- `ok` once the bidirectional
-/// stream is established, `error` when the client could not be built.
+/// A scout stream connection attempt resolved -- `ok` once the bidirectional
+/// stream is established, `error` when the client could not be built or the
+/// stream handshake with the API failed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/scout/src/metrics.rs` around lines 77 - 90, Update the doc comment for
ScoutStreamConnection to state that the error outcome covers both client
construction failures and scout_stream handshake failures. Keep the existing
description of successful bidirectional stream establishment and the metric
definition unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/scout/src/metrics.rs`:
- Around line 77-90: Update the doc comment for ScoutStreamConnection to state
that the error outcome covers both client construction failures and scout_stream
handshake failures. Keep the existing description of successful bidirectional
stream establishment and the metric definition unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3a0e4695-445e-48e3-8057-4a2354f7b72a

📥 Commits

Reviewing files that changed from the base of the PR and between eb4ea22 and e2ee040.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (5)
  • crates/scout/Cargo.toml
  • crates/scout/src/cfg/command_line.rs
  • crates/scout/src/main.rs
  • crates/scout/src/metrics.rs
  • crates/scout/src/stream.rs

@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown

🔍 Container Scan Summary

Service Total Critical High Medium Low Other
boot-artifacts-aarch64 3 0 0 3 0 0
boot-artifacts-x86_64 3 0 0 3 0 0
forge-admin-cli-x86_64 259 15 31 78 7 128
machine-validation-runner 809 48 238 290 36 197
machine_validation 809 48 238 290 36 197
machine_validation-aarch64 809 48 238 290 36 197
nvmetal-carbide 809 48 238 290 36 197
TOTAL 3501 207 983 1244 151 916

Per-CVE detail lives in the per-service grype-* artifacts (JSON + SARIF). Severity counts only — no CVE IDs published here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant